Frequency-Based Matching in the Fellegi-Sunter Model of Record Linkage

1. INTRODUCTION As a special case of their general theory of record linkage, Fellegi and Sunter (1969) presented a formal model for matching that uses the relative frequency of strings being compared. For instance, a surname that is relatively rare in pairs of records taken from two files has more distinguishing power than a common one. Most applications of frequency-based matching have used close variants of the basic model but have made different simplifying assumptions that reduce computation and facilitate table building. This paper introduces an extended methodology under weaker assumptions. While the amount of computation is significantly increased (as much as an order of magnitude), the need for expert human intervention is reduced. Most or all of the matching parameters can be automatically computed using file characteristics alone. The methodology does not require calibration data sets on which true match status has been determined. No a priori assumptions about parameters or previously created lookup tables are needed. Relative frequency tables are more suitable for situations when one list cannot be assumed a near subset of another. When one list is a near subset of the other and a number of other simplifying assumptions are made, the new method yields tables comparable to those obtained via previous methods. If the matching is performed on a subset of pairs (such as those agreeing on Soundex code of surname or on specific geographic identifiers), then adjustments of the parameters and decision rules to the subsets are also automatic. In the second section of the paper, background on the Fellegi-Sunter model of record linkage is presented. The third section is divided into five parts. The first contains the basic theory for the new frequency-based methods. The theory holds for all pairs in the product space of two files. In the second, a method of adjusting for typographical variation is given. The method partially accounts for the fact that observed frequencies do not necessarily correspond to true frequencies. The third part shows how matching decision rules can utilize both frequency-based weights and simpler agree/disagree weights obtained via the Expectation-Maximization (EM) Algorithm (Winkler 1988, 1989a,c; Thibaudeau 1989). As the EM-derived weights are sometimes obtained on subsets of pairs such as those agreeing on geographical subregions, two methods for adjusting the frequency-based weights to subsets are given. The fourth part contains empirical results for a comparison of files having substantial amounts of accurate information. In the …